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a Abstract. - We study the phase diagram of ultracold bosons in an optical lattice near a Fes- 
hbach resonance. Depending on the boson density, the strength of the optical lattice potential 

1-^ I and the detuning from resonance, the ground state can be a Mott insulator, a superfluid phase 

(~| . with both an atomic and a molecular condensate, or a superfluid phase with only a molecular 

Q ' condensate. Mott insulator to superfluid transitions can be induced either by decreasing the 

O I strength of the optical lattice potential or by varying the detuning from the Feshbach resonance. 

■ Quite generally, we flnd that for a commensurate density the ground-state may undergo several 

^isi ' insulator-superfluid or superfluid-insulator transitions as the magnetic fleld is varied through 

^ , the resonance. 

a^: 
m . 

^sr I Recent experiments on ultracold trapped atomic gases have opened a new window onto 

_j. ■ the phases of quantum matter [1] . A gas of bosonic atoms in an optical or magnetic trap has 

f^ I been reversibly tuned between superfluid (SF) and insulating ground states by varying the 

"Y^ ■ strength of a periodic potential produced by standing optical waves. This transition has been 

Cd I explained on the basis of the Bose-Hubbard model with on-site repulsive interactions and hop- 

^ ■ ping between nearest neighboring sites of the lattice [2]. As long as the atom- atom interactions 

are small compared to the hopping amplitude, the ground state remains superfluid. In the 

opposite limit of a strong lattice potential, the interaction energy dominates and the ground 

Q I state is a Mott insulator (MI) when the density is commensurate, with an integer number of 

O ■ atoms localized at each lattice site. Another aspect of trapped bosonic atoms that has been 

^ I studied extensively in recent years (without the optical lattice) is the dramatic increase in 

•'-j ■ the effective atom-atom interaction due to the presence of a Feshbach resonance [3,4]. Near 

rN I the resonance, which can be experimentally realized by tuning an external magnetic fleld, the 

j3 ■ atoms can resonantly collide to form molecular bound states. Remarkable phenomena asso- 

■ ■ ' ciated with a Feshbach resonance, including coherent atom-molecule oscillations and collapse 

of the Bose-Einstein condensate, have been experimentally observed [4]. 

In this Letter, we study a Bose gas in the presence of an optical lattice near a Feshbach 
resonance. Starting from an effective Bose-Hubbard Hamiltonian, we flrst obtain the various 
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Mott insulating states that are stable in the limit of a strong optical lattice. We then study 
the stability of these states by deriving an effective Landau-Ginzburg theory for the MI- 
SF transition. Depending on the boson density, the strength of the optical lattice potential 
and the detuning from resonance, the ground state can be a MI, a SF phase with both an 
atomic and a molecular condensate, or a SF phase with only a molecular condensate. MI-SF 
transitions can be induced either by decreasing the strength of the optical lattice potential or 
by varying the detuning from the Feshbach resonance. For odd commensurate densities, Mott 
phases are found to be always unstable to superfluidity on the molecular side of the Feshbach 
resonance. For even commensurate densities, Mott phases can be stable on both sides of the 
resonance and may undergo several MI-SF or SF-MI transitions as the magnetic field is varied 
through the resonance. 

Bosons in an optical lattice near a Feshbach resonance can be described by the generalized 
Bose-Hubbard Hamiltonian 

(J— a,?n (r.r') r 

r 

+C/a™^V'l(r)V'j„(r)V'™(r)^a(r) -a^(^1„(r)^„(r)^„(r) +h.c.). (1) 

r r 

ipcTftpt are bosonic operators for atoms (cr — a) and molecules {a = m). The discrete variable 
r labels the different sites (i.e. minima) of the optical lattice, (r, r') denotes a sum over 
nearest sites. The optical lattice is assumed to be bipartite with coordination number z. i^ is 
the intersite hopping amplitude, and Ua the amplitude of the atom-atom {a = a) or molecule- 
molecule {a = m) interaction. Uam gives the amplitude of atom-molecule interactions. /Xa = A* 
and fim — 2^, — v are the chemical potential for atoms and molecules. The chemical potential 
ji fixes the total number of atoms (free or bound into molecules) . u is related to the energy of 
a molecular bound state and is a function of the appHed magnetic field. A negative detuning 
from resonance {v < 0) favors the formation of molecules. The last term in Eq. (QJ describes 
the atom-molecule conversion and the parameter a depends on the resonance width [3,4]. 
The Hamiltonian QJ can be derived from a microscopic model starting from the bare atom- 
molecule Hamiltonian in the presence of the optical lattice [4] and expanding the operators 
i'a^i'm on the lowest-band Wannier states [5]. We shall not attempt to calculate the explicit 
values of Ua,trj, Uam and a from the microscopic Hamiltonian, but rather treat them as free 
parameters. We take Ua — Um — 1 and a = 0.5, and vary Uam and ta = tm since the 
qualitative aspects of the phase diagram do not depend on the precise values of a, Ua/Um and 
tm/ta [6]. We also neglect the trap potential, which is not expected to change qualitatively 
the main conclusions of our work. 

Let us first discuss the phase diagram in the "local" (i.e. single-site) Hmit where ta = 
tm = 0. For an integer number no of particles per site, the Hilbert space is spanned by 
the states \na,nm) = ('^a!"'m!)~^^^('/'l)"°(V'm)""k9'C) where the number of atoms (tIq) and 
molecules (um) satisfy Ua + 2nm — ng. |vac) denotes the vacuum of particles. In this basis, 
the Hamiltonian _//("") is a real symmetric matrix, defined by 

-f^n1°n„ = -^J■no + lyn„^ + -;^naina-l) + ^nm{nm.-l) + Ua„inanm, 
Ht:U,n„. - -a[ina + 2)ina + l)n^]'/^ (2) 
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Fig. 1 - Phase diagram in the local limit (fa = tm = 0) vs chemical potential /i and detuning u from 
the Feshbach resonance, a — 0.5, Ua ~ Um ~ 1. Uam = 1 (left), Uam = 0.25 (right). Each phase is 
labeled by the number no of atoms (free or bound into molecules) per site. 



which can be numerically diagonalized. In the following we shall denote the eigenstates and 
eigenenergies by {(f)^' , £1"^°'), i = 1, • • ■ , dng, where d„o = no/2 + 1 ((no + l)/2) for no even 
(odd). 

The phase diagram in the local limit as a function of the chemical potential fi and the 
detuning u is shown in Fig.Q]for Uam = 1 and Uam = 0.25. Each phase is labeled by the total 
number no = na + 2nm of bosons per site where na and Um are the mean number of atoms 
and molecules. First let us consider the Mott states for large detuning |i^|/a 3> 1. For even 
no, the Mott states are stable on both sides of the resonance: for i^ > 0, there are n^ — uq 
atoms per site whereas for z^ < there are n^ — no/2 molecules per site. For no odd, the 
Mi's are stable only for u > 0. For ly < 0, the ground state for odd no is a superposition 
of two Mott phases with (no — l)/2 and (no + l)/2 molecules per site and hence unstable to 
superfluidity for any flnite ta,tm- The phase diagram for small detuning (|j^|/a < 1), on the 
other hand, depends crucially on Uam- In contrast to the case Uam > 1, for Uam ^ 1, the 
Mott phases with odd no can occur even when ;/ < as seen in Fig.^ 

In order to study the instability of the Mott phases towards superfluidity in the presence 
of flnite intersite hopping, we perform an expansion about the local Hmit. Our approach is 
a generalization of mean-fleld theories previously used for the Bose-Hubbard model [7]. We 
write the partition function Z as a functional integral over complex bosonic flelds 'ipa,4'm 
with the action SltpaTipm] = /g dT{J2r (7'^ai''^)9ri'cr{''^) + H[^]^ (r is an imaginary time and 
P — 1/T the inverse temperature). Introducing an auxihary fleld 0o- to decouple the intersite 
hopping term by means of a Hubbard-Stratonovich transformation, we obtain 



z = yp[^,</)]e 



-(0|i-V) + ((0l'/')+c.c.)-SoW 



Zo / V[(t>]e 



V[cl>]e 



'(01*^ 0)/p(0IV')+c.c. 



)o 



^(0|t-i0)+W'[0] 



(3) 



where we use the shorthand notation (0|V') = /q '^'''X^r o- '^o-(i")V'(T(r), etc. t~^ denotes the 
inverse of the intersite hopping matrix deflned by i^r' = ^^ if ^- ^' ^^'6 nearest neighbors and 
irr' = otherwise. 5*0 is the action of the -0 field in the local Hmit {ta = tm = 0). (■ • ■)o 
means that the average is taken with 5*0 [V']- In the last Hne of lO, we have introduced the 
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Fig. 2 - Mean-field phase diagram obtained from the free energy F [Eq. ^]. MI: Mott insulator, 
AC-MC: SF phase with AC and MC, MC: SF phase with MC. The solid (dashed) lines indicate second 
(first) order transitions. T = (0, A2(g„/ga)^''V[4(flaffm)^''^+5am]) and J = (AV[4(3a3m)^''^+Sfam],0) 
separate second and first order MI-SF transition lines. 



generating functional of connected local Green's functions [8]: 
+ Y. Gli^(l,2;3,4)0:(l)0;,(2)0™(4)0,(3) 

1,2,3,4 

+ J E [^-™(l' 2; 3)</>:(l)0:(2)0„(3) + c.c], 



(4) 



1,2,3 



up to quartic order in the fields. Here i = ivi.Ti) and X^i = Jo ^'^i^x ■ Ga(l,2) = 
-{V'.(l)C(2))o,G^^^(l,2;3,4) = (V'.(l)^.(2)C(4)V;(3))o,c and Gii^(l, 2;3,4) = (V;,(l)V;™(2)^;;(4)V':(3))o,c 
are single-particle and two-particle connected local Green's functions, and Aam(l,2;3) = 

(V'a(l)^a(2)V:.(3))o. 

We determine the zero-temperature phase diagram within a mean-field approximation 
where 0a and 0™ are assumed to be time and space independent. A finite value of ^o- signals 
superfiuidity since (/)(j = ztc{'4'a) at the mean-field level. The free energy (per site) F is 
obtained to quartic order from Eqs. IJ3I4|I : 



F — Fq 



+ E('''^'^^ + Y^v ^ ■9'^™'^^'^™ ~ ^'^^'^' 



(5) 



where, with no loss of generality, the order parameters can be chosen real. Fo = — TlnZo is 
the free energy in the local limit. The coefficients of the free energy expansion Q are given 
by Ta — (zta)^^ + Ga, 5ct — —G]^'^/2, gam = — G^m, and A — Aam where all the local Green's 
functions are evaluated in the static limit and for T — > 0. For instance Go- = Ga{iuj = 0), 
where G. (iuj) is the Fourier transform of G. (t) {lu is a bosonic Matsubara frequency) . The 
mean-field phase diagram obtained from F is shown in Fig. [21 [9]. There are three different 
ground states: MI: 0a = 0m = 0, SF phase with both an atomic and a molecular condensate 
(AC-MC): (j)a,4>m 7^ 0, and SF phase with only a molecular condensate (MC): 0a — 0,(j>m ^ 
0. The existence of two different SF phases in a Bose-Einstein condensate near a Feshbach 
resonance has been recently predicted in Ref. [10] [11]. Both SF phases spontaneously break 
the U(l) gauge invariance, i.e. the invariance under the phase transformations 0a -^ 4>ae'^^ and 
0m -^ 0me^'^. The MC phase breaks only the U(l)/Z2 symmetry since 0m -^ (pm for 9 = n. 
The residual discrete Z2 symmetry is spontaneously broken at the (Ising-like) transition to 
the AC-MC phase which occurs near the resonance [10]. 
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Fig. 3 - Same as Fig. U but for a finite intersite hopping zta ~ ztm = 0.04. The Mi's (labeled by 
the number no of atoms (free or bound into molecules) per site) are surrounded by SF phases. The 
solid (dashed) lines indicate a transition from a MI to an AC-MC (MC) SF phase. Transition lines 
between AC-MC SF phases and MC SF phases are not shown. 



The phase transitions are second order except near Va = rm = where they become first 
order (Fig.|2ll [12]. In the following, we neglect the possibility of first-order phase transitions so 
that the phase boundaries of the Mott insulating phases are determined by Ta = or r™ = 0. 
This approximation overestimates the stability of the Mott phases, but is expected to give a 
correct qualitative description of the phase diagram. Since the region where first-order phase 
transitions occur shrinks with A (and therefore a), as shown in Fig. [2l this approximation 
should be quantitatively correct when a <C Ua,Um, Uam- 

To proceed further, we need to determine r^ and therefore the single-particle local Green's 
function G^. Using the eigenstates </>,^"°^ of Jj("''\ we find (for T = 0) 



Gct 



E 

Z=l 



|(</,f°-''-V.|4"°^)P 



E. 



(no-Pa) 



E' 



("o) 



+ 



E 

i=l 



I (01"" V. 10. 



(no+Pa) 



w 



E 



("o) 



K 



{no+p^) 



(6) 



where Pa — I and Pm ~ 2, and 



lino) 



denotes the ground state of H'-"'°> . Fig. El shows the 



phase diagram in the (z^, /i) plane for zta — ztm — 0.04. Mott phases with different np are 
now separated by SF phases. Transitions between AC-MC and MC phases, which are beyond 
the scope of this Letter, are not shown. Mi's with odd and even commensurate densities 
(to be referred to as "odd" and "even" Mi's in the following) exhibit different behaviors. As 
ta — tm increases, odd Mi's become instable towards an AC-MC SF phase and are pushed 
to higher values of ly. Above the critical value zta = Ua{2no + 1 + V"i + "^)^^' the odd 
MI with no atoms per site has become entirely superfiuid. The behavior of even Mi's is more 
involved. They can be stable on both sides of the Feshbach resonance, and are unstable both 
towards AC-MC and MC phases as p, /x or i^ = tm are varied. The phase diagram and 
the disappearance of the Mi's as ta — tm increases turns out to be strongly dependent on 
the precise values of Uam and a, as well as on Ua/Um and ta/tm- Quite generally, however, 
we find that for a commensurate density and certain values of the strength of the optical 
lattice the ground-state may undergo several MI-SF or SF-MI transitions as f is varied. This 
phenomenon always occurs for even commensurate densities [see Fig. for no = 4 (left panel) 
and no = 2 (right panel)] but is restricted to small values of Uam for odd commensurate 
densities [right panel of Fig. El for the no = 3 phase]. 

The difference between odd and even phases is also clearly seen by considering the phase 
diagram at constant density. The density n is given by n = —dF/dfx. In the SF phase near 
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Fig. 4 - Phase diagram for n — "i vs intersite hopping amplitude ta ~ tm and detuning v from the 
Feshbach resonance, a — 0.5 and Ua = Um ~ 1- Vam = 1 (left), Uam ~ 0.25 (right). 



the SF-MI transition, we obtain 



no 



no 



Q'^a. 



dfi 






(AC - MC phase) 
(MC phase) 



(7) 



fi' a 



0, the 



where no is the density of the nearby MI. Thus at the transition points where i9, 
density n remains commensurate and equal to no. The condition d^jj^ — impHes that the 
boundary between the MI and the SF phases has a vertical tangent in the (y, /i) plane (Fig.EJ. 
The phase diagrams for n — i and n = 2 are shown in Figs. 21 and As discussed above, 
we observe MI-SF and SF-MI transitions as v is varied at constant ta = tm- For large Uam, 
the MI will be more robust for an integer mean number of atoms and molecules, whereas for 
small Uam it is more likely to become superfluid. This explains why the behavior of the n = 2 
phase near ly ^ 1 is strongly dependent on Uam (Fig. EJ . 

Let us finally discuss a few important points with respect to experiments. In the vicinity 
of the Feshbach transition, where the interaction energy becomes strong, the vaHdity of a 
single-band Hubbard model becomes questionable. However, this effect is significant only in 
a region of width a/U around i/ = and will not change our conclusions for most regions of 
the phase diagram for narrow resonances. Quantitative predictions of our model have to be 
supplemented with standard T-matrix renormalization effects near the MI-SF transition. This 
requires a more thorough analysis of the role of the optical lattice [13] and is beyond the scope 
of our work. However, we expect the qualitative predictions of the present theory regarding 
the Mott phases and the MF-SI transitions to be valid. At high lattice fiUing, molecules are 
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Fig. 5 - Same as Fig.^Jbut for n — 2. The solid (dashed) lines indicate a transition from a MI to an 
AC-MC (MC) SF phase. 
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likely to be unstable due to inelastic three-body collisions; this difficulty does not arise in the 
n = 2 and n = 1 Mott phases, which are therefore good experimental candidates to test our 
predictions. In the presence of the trap potential, superfluid and Mott phases can coexist in 
certain ranges of experimental parameters, and MI-SF transitions should rather be seen as 
crossovers than real quantum phase transitions [14]. However, this does not invalidate our 
main conclusions as stated below (if we understand transitions as crossovers) . 

In conclusion, we have shown that a Bose gas in an optical lattice near a Feshbach reso- 
nance exhibits a very rich phase diagram. Characteristic features of this phase diagram, which 
are qualitatively robust irrespective of precise numerical parameter values, are: i) MI-SF tran- 
sitions can be induced either by decreasing the strength of the optical lattice or by varying the 
magnetic field which controls the detuning from resonance, ii) Mi's with odd commensurate 
density are always unstable to superfluidity on the molecular side of the resonance whatever 
the strength of the optical lattice potential, iii) Mi's with even commensurate density can 
be stable on both sides of the resonance, iv) For a commensurate density the ground-state 
may undergo several MI-SF or SF-MI transitions as the magnetic field is varied through the 
resonance. We therefore expect a measurement of the momentum distribution function [1] 
to display several transitions between peaked interference patterns (SF state) and featureless 
uniform distribution (MI) as the magnetic field is varied. Another possible experiment would 
be to measure the evolution of the excitation spectrum as the magnetic field is varied, using 
two-photon Bragg spectroscopy [15]. 

K.S. thanks S.M. Girvin for support during completion of this work. 
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